Skip to content

Compile installed packages and pytest's plugins when the image is built (~2.7x faster) - #128

Merged
BethanyG merged 3 commits into
mainfrom
precompile-bytecode
Nov 18, 2026
Merged

BethanyG merged 3 commits into
mainfrom
precompile-bytecode

Conversation

@iHiD

@iHiD iHiD commented Nov 18, 2026 •

Copy link
Copy Markdown
Member

Hello 🙂 I'm working with Claude to try and get our tooling faster.

This PR makes this project around 2.7x faster. I've run it in production to benchmark things in the real world against the most recent few hundred solutions, so I'm confident that it works robustly. However, I've not written the code. I have tried my hardest to get Claude to write sensible messages and comments that explains stuff, but I'd ask for your patience in the fact I'm sending AI code to you. Everything below here is from your friendly neighbourhood LLM...


What changed

The image now includes precompiled .pyc files for pytest and its dependencies. The Dockerfile used to install the packages with --no-compile and PYTHONDONTWRITEBYTECODE=1, so the image had no .pyc files in site-packages. Every run compiled pytest and its dependencies from source before running any tests, and the compiled files were lost when the container stopped. As suggested in review, pip now compiles the packages as it installs them, and compileall compiles only the runner (with --invalidation-mode=checked-hash).

The image also caches pytest's rewritten plugins. pytest rewrites the assert statements in its plugins (such as pytest-subtests) when it imports them, and caches the rewritten copies separately from normal bytecode. A new step, also suggested in review, runs pytest --collect-only once on a dummy test when the image is built, so each run loads those cached copies instead of rewriting the plugins again. That step unsets PYTHONDONTWRITEBYTECODE, because pytest won't write the files while it's set.

The image grows from about 59 MB to 70 MB.

This has been tested in production, and has reduced the median run from 2.5s to 0.9s.

🤖 Generated with Claude Code

The packages are installed with --no-compile and PYTHONDONTWRITEBYTECODE, so
the image held no .pyc files for pytest or its dependencies. Every run
compiled them from source before it ran any tests. Compiling them once at
image build takes about 40% off each run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@iHiD iHiD changed the title Compile installed packages to bytecode when the image is built Compile installed packages to bytecode when the image is built (~40% faster) Nov 18, 2026
@BethanyG

Copy link
Copy Markdown
Member

Thanks for this. I think the failing CI tests have to do with the Python version mis-match and error cases. The line number changes per python version. I will see if its an easy re-gen fix.

According to what I've read, Standard precompilation via python -m compileall misses assertion rewriting for pytest. So when pytest hits an import, it will recompile anyways. The recommendation is to use Run pytest --collect-only, which will trigger pytest's actual assertion rewriter, caching the exact .pyc files needed for test execution.

But I do wan tot dig a little further, so please bear with me. 🙂

pytest rewrites the assert statements in its plugins (such as
pytest-subtests) when it imports them, and caches the rewritten copies
separately from normal bytecode, so compileall can't create them. Running
pytest once with --collect-only on a dummy test at image build creates
those cached copies, so each run loads them instead of rewriting the
plugins again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@iHiD iHiD changed the title Compile installed packages to bytecode when the image is built (~40% faster) Compile installed packages and pytest's plugins when the image is built (~2.7x faster) Nov 18, 2026
@iHiD

iHiD commented Nov 18, 2026

Copy link
Copy Markdown
Member Author

Thanks for looking at this, Bethany 🙂 This reply is from Claude, on Jeremy's behalf.

You're right that compileall doesn't produce pytest's assertion-rewritten files. The original saving is real, though, because it comes from pytest itself. The packages are installed with --no-compile, so the image had no .pyc files in site-packages at all. Every run compiled pytest, its dependencies and the parts of the standard library they use from source, before it touched the student's code. compileall removes that work, and on its own it cut the median run by about 40%.

Your --collect-only suggestion turned out to help a lot on top of that, so thank you! It caches the rewritten copies of the plugins (pytest-subtests and platformdirs). I benchmarked it on 20 example solutions, and it took the median run down by about another 40%. The results were identical, so I've added it to this PR as a second commit. Together the two steps make a run about 2.7x faster.

On CI: the failing example-syntax-error test compares a traceback that includes line numbers inside Python's frozen importlib. The base image uses a floating python:3.13 tag, so a fresh build picks up a newer patch release with different line numbers. Main fails the same way when it's rebuilt today, so regenerating that golden file should fix it.

@BethanyG

BethanyG commented Nov 18, 2026 •

Copy link
Copy Markdown
Member

On looking at this further, I think we could make the following changes, and maybe avoid having to compile the entire site-pacages directory in a separate step, since pip will compile what's needed during install. Doesn't get us out of compiling the runner or runner scripts, tho.

ENV PYTHONDONTWRITEBYTECODE=1 \
    PYTHONUNBUFFERED=1

RUN --mount=type=cache,target=/root/.cache/pip\
 pip install -r /requirements.txt

Pip will automatically look for a wheel and compile while installing (unless prevented by the --no-compile flag), so that will take care of pytest, black, and all dependancies. This may or may not be faster than just compiling site-packages wholesale.

Another suggestion:


RUN python3 -m compileall -q --invalidation-mode=checked-hash \
    /opt/test-runner/runner /opt/test-runner/bin

The --invalidation-mode=checked-hash ensures that timestamp differences don't invalidate things across builds.

@ihid-assistant

Copy link
Copy Markdown

Thanks Bethany, these are great suggestions! Letting pip compile during the install is simpler than a separate step over the whole of site-packages, and it keeps the Dockerfile closer to how it was.

I'll benchmark your version against this PR on the same 20 example solutions: pip compiling during the install, compileall over just the runner with --invalidation-mode=checked-hash, and the --collect-only warm-up kept. I'll post the timings and image sizes here so you can choose which you'd prefer.

Pip now compiles the packages while installing them, and compileall only handles the runner (with checked-hash files). The pytest warm-up unsets PYTHONDONTWRITEBYTECODE so it can still write its rewritten plugin files.

Co-authored-by: iHiD <286476+iHiD@users.noreply.github.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ihid-assistant

Copy link
Copy Markdown

Thanks again, Bethany. I benchmarked your approach, and it's the best version, so I've switched the PR to it.

Letting pip compile during the install is as fast as compiling all of site-packages separately, and the image is about 6 MB smaller. Over all 142 example solutions, the median run went from 2.33s on main to 0.85s, with identical results.

There's one small adjustment. With PYTHONDONTWRITEBYTECODE=1 set in the image, the --collect-only warm-up can't write pytest's rewritten plugin files, and each run then takes about twice as long (1.63s). So the warm-up step unsets that variable for itself. The invalidation mode made no measurable difference either way, so I've kept your checked-hash.

The only failing golden test is still example-syntax-error, the same as on main.

@BethanyG

Copy link
Copy Markdown
Member

Thank you for the changes and the benchmark numbers!

The only failing golden test is still example-syntax-error, the same as on main.

Since we well know why that's failing (and it is getting late for me) - I am going to go ahead and merge this so we can take advantage of the speed bump-up. 😄

@ihid-assistant

Copy link
Copy Markdown

Thank you, Bethany! Your suggestions made this both faster and simpler than where it started, so thanks for digging into it. I'll check the production run times once it's deployed.

@BethanyG
BethanyG merged commit 4e23039 into main Nov 18, 2026
1 of 2 checks passed
@BethanyG
BethanyG deleted the precompile-bytecode branch November 18, 2026 10:22
@ihid-assistant

ihid-assistant Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

Thanks again for merging this, @BethanyG, and for your suggestions. Here are the production numbers since it was deployed at 10:22 UTC.

Median p95
Before (05:00 to 10:00 UTC) 2.45s 3.9s
After (11:00 UTC onwards) 0.89s 1.4s

The "after" figures cover about 300 runs. Most of the gain comes from your idea of warming pytest with --collect-only at image build. Without it, the PR would only have reached about 1.5s. Your switch to letting pip compile during the install also kept the image smaller. So Python runs in production are now nearly 3x faster, and the slowest 5% are faster still. Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants